Method and system for context-aware decision making of an autonomous agent

ABSTRACT

A system for context-aware decision making of an autonomous agent includes a computing system having a context selector and a map. A method for context-aware decision making of an autonomous agent includes receiving a set of inputs, determining a context associated with an autonomous agent based on the set of inputs, and optionally any or all of: labeling a map; selecting a learning module (context-specific learning module) based on the context; defining an action space based on the learning module; selecting an action from the action space; planning a trajectory based on the action S260; and/or any other suitable processes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/846,870, filed 22 Jun. 2022, which is a continuation of U.S.application Ser. No. 17/584,062, filed 25 Jan. 2022, which is acontinuation of U.S. application Ser. No. 17/332,839, filed 27 May 2021,which is a continuation of U.S. application Ser. No. 17/306,014, filed 3May 2021, which is a continuation of U.S. application Ser. No.17/116,810, filed 9 Dec. 2020, which claims the benefit of U.S.Provisional Application Ser. No. 63/035,401, filed 5 Jun. 2020, and U.S.Provisional Application Ser. No. 63/055,756, filed 23 Jul. 2020, each ofwhich is incorporated in its entirety by this reference.

TECHNICAL FIELD

This invention relates generally to the autonomous vehicle field, andmore specifically to a new and useful system and method forcontext-aware decision making of an autonomous agent in the autonomousvehicle field.

BACKGROUND

In autonomous and semi-autonomous control of automotive vehicles,conventional systems and methods for decision making can be classifiedas one of two approaches: classical approaches relying on traditionalprogramming principles and machine learning based approaches. However,each of these approaches comes with its limitations. Further, a largepercentage of current autonomous vehicle systems and methods attempt todrive in various different environments, which makes either theclassical approaches extremely involved (and most likely impossible) orthe machine learning based approaches lacking explainability (andtherefore causing safety concerns).

Thus, there is a need in the autonomous vehicle field to create animproved and useful system and method for decision making.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic of an autonomous vehicle system for contextselection.

FIG. 2 is a schematic of a method for autonomous vehicle contextselection.

FIG. 3 depicts a variation of an integration of the system for contextselection (context selector) within an autonomous vehicle architecture.

FIG. 4 depicts a variation of a set of contexts.

FIG. 5 depicts a variation of a vehicle executing a creep behaviorwithin a particular context.

FIG. 6 depicts a schematic of a variation of context switching for fixedroutes in comparison to the route options in a geo-fenced operationaldesign domain (ODD).

FIGS. 7A-7D depict an example of context selection along a route.

FIG. 8 depicts a schematic variation of an overall system of theautonomous agent.

FIG. 9 depicts a schematic variation of context-aware decision makingand trajectory planning.

FIGS. 10A-10B depict a variation of a use case of an autonomous vehiclein fixed-route deliveries and a schematic of fixed routes driven by thevehicles.

FIG. 11 depicts a variation of an integration of a context selector in atrajectory generation architecture.

FIG. 12 depicts a variation of a context selector in which a context isselected based on a map and a location and/or orientation of thevehicle.

FIG. 13 depicts a variation of a context identifier module whichdetermines a context for the vehicle based on a set of inputs and one ormore models and/or algorithms.

FIG. 14 depicts a variation of a map indicating a sequential series ofcontexts for each of a set of example routes.

FIG. 15 depicts a variation of map indicating an example set of contextregion assignments.

FIG. 16 depicts a variation of a context selector and set of learneddeep decision networks.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the inventionis not intended to limit the invention to these preferred embodiments,but rather to enable any person skilled in the art to make and use thisinvention.

1. Overview

As shown in FIG. 1, a system 100 for context-aware decision making of anautonomous agent includes a computing system having a context selectorand a map. Additionally or alternatively, the system can include and/orinterface with any or all of: an autonomous agent (equivalently referredto herein as an autonomous vehicle and/or an ego vehicle); a vehiclecontrol system; a sensor system; and/or any other suitable components orcombination of components.

As shown in FIG. 2, a method 200 for context-aware decision making of anautonomous agent includes receiving a set of inputs S210 and determininga context associated with an autonomous agent based on the set of inputsS220. Additionally or alternatively, the method 200 can include any orall of: labeling a map S205; selecting a learning module(context-specific learning module) based on the context S230; definingan action space based on the learning module S240; selecting an actionfrom the action space S250; planning a trajectory based on the actionS260; and/or any other suitable processes. The method 200 can beperformed with a system as described above and/or any other suitablesystem.

2. Benefits

The system and method for context-aware decision making of an autonomousagent can confer several benefits over current systems and methods.

First, in preferred variations, the system and/or method confers thebenefit of decision making through a hybrid approach of deep learningand rule-based processes, achieving explainable artificial intelligence(AI).

This can in turn confer the benefit of achieving a safe and scalablesolution for autonomy by any or all of: decomposing learned models intomicro-modules with intended functionality restricted to very explainabletasks; building rule-based fallback and validation systems around themicro-modules to guarantee safety, which enables validation of theperformance and underlying properties of each of these sub-modules;and/or any other suitable processes.

Second, in preferred variations, the system and/or method furtherconfers the benefit of reducing the amount of data required to traverse,validate, and/or add a new route by implementing a small, limited-routeODD including a small number of contexts which can be overly definedand/or described, thereby eliminating and/or reducing a number of edgecases encountered by the vehicle. Equivalently, the system and/or methodcan reduce the need for a large amount of data due to acute confinementof problem while maintaining all the benefits of learning systems (e.g.,maintaining an extremely low margin of error in decisions, enablinghuman-style driving decisions, continued driving progress, etc.). Thiscan enable an optimal selection of learning modules (e.g., deep learningmodels) and/or training of the learning modules based on low requireddata and or minimal edge cases; in some examples, for instance, inversereinforcement learning algorithms (which conventionally require diverseand significant amounts of data to be trained) can be leveraged, whicheffectively replicate human driving. In specific examples, significantlyless data (e.g., 50 times less, 100 times less, 1000 times less, etc.)is required than geofenced approaches to validate a route.

Third, in some variations (e.g., as shown in FIGS. 10A-10B), the systemand/or method confers the benefit of enabling supply chain growth inshort haul logistics (e.g., B2B trucking) applications, by enablingfixed route autonomous delivery of goods between locations.

Fourth, in preferred variations, the system and/or method furtherpreferably confer the benefit of enabling determination and awareness ofa context of an autonomous vehicle, which can confer the subsequentbenefits of: achieving smooth switching of contexts (e.g., with minimalvalidation and testing, based on a predetermined series of contextsspecified for particular fixed route, etc.); training models based on aspecific context (therefore reducing the training required for eachmodel), such as optimizing for different cost functions and/orparameters in different contexts; selecting safe and appropriatebehaviors/actions for the vehicle in light of the context; and/or canconfer any other benefit.

Fifth, in some variations, the system and/or method confers the benefitof overcoming the shortcomings of conventional systems and methods,which can include, for instance, any or all of: an inability to adapt tounexpected situations; overgeneralization, which often leads toconservative behavior; hard-to-tune (e.g., hard to manually tune)parameters; constraint monitoring and adaptation of hyper parametersbased on environmental changes; unsolvability even in small ODDs; and/orany other shortcomings of conventional systems and methods.

Additionally or alternatively, the system and method can confer anyother benefit.

3. System 100

The system 100 functions to enable context selection and context-awaredecision making for a vehicle and includes: a computing system having acontext selector and a map. Additionally or alternatively, the systemcan include and/or interface with any or all of: an autonomous agent; avehicle control system; a sensor system; and/or any other suitablecomponents or combination of components.

The system 100 is preferably configured to implement and/or interfacewith a system which implements a hybrid architecture of decision making(e.g., as shown in FIG. 3, as shown in FIG. 8, as shown in FIG. 9, asshown in FIG. 11, etc.), the hybrid architecture implementing bothclassical, rule-based approaches and machine learning approaches, whichis enabled by a small ODD, fixed route driving framework. This functionsto maintain explainability of the vehicle's decision making whileenabling the vehicle to drive with human-like driving behavior on routesvalidated with minimal training data.

In a first set of variations, as shown in FIGS. 10-10B, the system 100is implemented in autonomous short-haul (e.g., between 5 and 400 miles)B2B fixed-route applications. In these variations, the autonomous agentspreferably receive inventory from sorting centers, but can additionallyor alternatively receive inventory for parcel hubs and/or warehouses.The agent then preferably delivers the inventory to and/or between anyor all of: sorting centers, micro-fulfillment centers, distributioncenters, retail stores, and local delivery centers. Additionally oralternatively, the agents can interface with residences (e.g., customerhomes), and/or any other suitable locations/facilities.

3.1 System—Components

The system 100 includes a computing system, which functions to select acontext associated with the vehicle. Additionally or alternatively, thecomputing system can function to perform any or all of: route planningof the vehicle at a planning module (e.g., generating a trajectory);localization of the vehicle and/or surrounding objects at a localizationmodule; path prediction of the vehicle and/or objects surrounding thevehicle at a prediction module; storage of information; and/or any othersuitable functions.

The computing system is preferably designed to offer a centralized andparallel computing model which enables high concurrency of taskexecution, low latency, and high throughput. The adaptive communicationcapabilities of the framework allows for high data throughput while theuser-level scheduler with resource awareness enables the centralizedcomputing model to perform at the highest level.

To enable this, the computing system is preferably designed at leastpartially in a modular format including a set of modular computingcomponents, referred to herein as learning modules (equivalentlyreferred to herein as learning agents or learning models), eachassociated with predefined inputs and outputs. Each computing componentcontains a specific algorithm module built to process a set of datainputs and generate a set of outputs. The computing system canoptionally include a middleware framework, which extracts dependenciesfrom these components and links them all together (e.g., with atopological ordering process such as a directed acyclic graph, etc.). Atruntime, for instance, the framework takes the predefined componentsdescribed above and combines them with fused data from the sensors tocreate lightweight user-level tasks. Each task is then scheduled basedon resource availability and task priorities and executed as optimizedthreads.

Additionally or alternatively, the system and/or computing system can beotherwise configured and/or designed.

The computing system includes an onboard computing system onboard (e.g.,integrated within) the autonomous agent.

In preferred variations, the autonomous agent includes an autonomousvehicle that is preferably a fully autonomous vehicle and/or able to beoperated as a fully autonomous vehicle, but can additionally oralternatively be any semi-autonomous or fully autonomous vehicle, ateleoperated vehicle, and/or any other suitable vehicle. The autonomousvehicle is preferably an automobile (e.g., car, driverless car, bus,shuttle, taxi, ride-share vehicle, truck, semi-truck, etc.).Additionally or alternatively, the autonomous vehicle can include any orall of: a watercraft (e.g., boat, water taxi, etc.), aerial vehicle(e.g., plane, helicopter, drone, etc.), terrestrial vehicle (e.g.,2-wheeled vehicle, bike, motorcycle, scooter, etc.), and/or any othersuitable vehicle and/or transportation device, autonomous machine,autonomous device, autonomous robot, and/or any other suitable device.

The computing system can additionally or alternatively include a remotecomputing system offboard the autonomous agent, such as a cloudcomputing system. The remote computing system is preferably incommunication with the onboard computing system (e.g., to collectinformation from the onboard computing system, to provide updated modelsto the onboard computing system, etc.), but can additionally oralternatively be in communication with any other suitable components.

The computing system preferably includes active and redundantsubsystems, but can additionally or alternatively include any othersuitable subsystems.

The computing system preferably includes a context selector, whichfunctions to select a context associated with the vehicle. The contextselector is further preferably part of a planning module of thecomputing system, which can additionally include any or all of: a set oflearning modules (e.g., deep learning models); a trajectory generator; atrajectory validator; and/or any other suitable components. Additionallyor alternatively, the context selector can be independent from aplanning module, a planning module can include any other suitablecomponents, and/or the computing system can be otherwise configured.

The computing system further preferably includes a processing system,which functions to process the inputs received at the computing system.The processing system preferably includes a set of central processingunits (CPUs) and a set of graphical processing units (GPUs), but canadditionally or alternatively include any other components orcombination of components (e.g., processors, microprocessors,system-on-a-chip (SoC) components, etc.).

The computing system can optionally further include any or all of:memory, storage, and/or any other suitable components.

In addition to the planning module, the computing system can includeand/or interface with any or all of: a localization module, predictionmodule, perception module, and/or any other suitable modules foroperation of the autonomous agent.

The computing system (e.g., onboard computing system) is preferably incommunication with (e.g., in wireless communication with, in wiredcommunication with, coupled to, physically coupled to, electricallycoupled to, etc.) a vehicle control system, which functions to executecommands determined by the computing system.

The computing system includes and/or interfaces with a map, whichfunctions to at least partially enable the determination of a contextassociated with the autonomous agent. The map is preferably a highdefinition, hand-labeled map as described below, which prescribes thecontext of the autonomous agent based on its location and/or positionwithin the map, but can additionally or alternatively include any othermap (e.g., map labeled in an automated fashion, map labeled through bothmanual and automated processes, etc.) and/or combination of maps.

The system 100 preferably includes and/or interfaces with a sensorsystem (equivalently referred to herein as a sensor subsystem), whichfunctions to enable any or all of: a localization of the autonomousagent (e.g., within a map), a detection of surrounding objects (e.g.,dynamic objects, static objects, etc.) of the autonomous agent, and/orany other suitable function.

The sensor system can include any or all of: cameras (e.g., 360-degreecoverage cameras, ultra-high resolution cameras, etc.), light detectionand ranging (LiDAR) sensors, radio detection and ranging (RADAR)sensors, motion sensors (e.g., accelerometers, gyroscopes, inertialmeasurement units [IMUs], speedometers, etc.), location sensors (e.g.,Global Navigation Satellite System [GNSS] sensors, Inertial NavigationSystem [INS] sensors, Global Positioning System [GPS] sensors, anycombination, etc.), ultrasonic sensors, and/or any suitable sensors.

In a set of variations, the sensor system includes: 16-beam LIDARs(e.g., for high fidelity obstacle detection, etc.); short range RADARs(e.g., for blind spot detection, cross traffic alert, emergency braking,etc.); ultrasonic sensors (e.g., for park assist, collision avoidance,etc.); 360-degree coverage cameras (e.g., for surround view forpedestrian/cyclist/urban obstacle detection and avoidance, etc.);128-beam LIDAR (e.g., for localization of vehicle with high precision);long range ultra-high resolution cameras (e.g., for traffic sign andtraffic light detection); long range RADARs (e.g., for long rangeobstacle tracking and avoidance); GNSS/INS (e.g., for ultra-highprecision localization); and/or any other suitable sensors.

In preferred variations, for instance, the sensor system is configuredto enable a determination of a pose (and/or any other location and/ororientation parameter(s)) of the agent, which is used to select acontext associated with that pose (e.g., in a 1:1 fashion, with multipleoptions for context, etc.) based on a labeled map. In specific examples,the sensor system includes a localization subsystem which determines thepose, wherein the localization subsystem includes any combination ofGPS, IMU, LIDAR, camera, and/or other sensors mounted on the vehicle toestimate its current position at any given time. The sensor systemfurther preferably enables the determination of information (e.g.,location, motion, etc.) of objects and/or features in the environment ofthe agent, such as, but not limited to: dynamic objects, static objects,road infrastructure, environmental conditions (e.g., rain, snow,lighting conditions, etc.), and/or any other suitable information.

Additionally or alternatively, the sensor system can include any othersuitable sensors configured to collect any suitable sensor information.

Further additionally or alternatively, the system 100 can include anyother suitable components or combination of components.

4. Method 200

As shown in FIG. 2, the method 200 includes receiving a set of inputsS210 and determining a context associated with an autonomous agent basedon the set of inputs S220. Additionally or alternatively, the method 200can include any or all of: labeling a map S205; selecting a learningmodule (context-specific learning module) based on the context S230;defining an action space based on the learning module S240; selecting anaction from the action space S250; planning a trajectory based on theaction S260; and/or any other suitable processes.

The method preferably implements and/or interfaces with a systemimplementing hybrid decision making as described above, but canadditionally or alternatively implement and/or interface with methodsimplementing any other suitable decision making.

The method 200 functions to determining a context associated with anautonomous agent and thereby enable context-aware decision making of theautonomous agent. Additionally or alternatively, the method 200 canfunction to enable selection of an action to be performed by theautonomous agent, the generation of a trajectory to be traveled by theautonomous agent, and/or perform any other suitable function(s).

The method 200 is preferably performed with a system 100 as describedabove, but can additionally or alternatively be performed with anysuitable system.

The method 200 is preferably performed throughout the duration of theroute being traveled by the autonomous agent and based on a map (e.g.,continuously checking for a context change), but can additionally oralternatively be performed at any or all of: a predetermined frequency(e.g., constant frequency), in response to a trigger, at a set ofintervals (e.g., random intervals), once, and/or at any other suitabletimes.

4.1 Method—Labeling a Map S205

The method 200 can optionally include labeling a map S205, whichfunctions to specify the context for the vehicle at any location of theautonomous agent along a route (e.g., predetermined route). Additionallyor alternatively, labeling the map can function to specify the locationof one or more static objects along a route of the agent; a locationand/or other information of lane boundaries and/or other road features(e.g., information about current lane, lane boundaries, lane lines,etc.); the locations of starting points and vehicle destinations; a setof parameters (e.g., speed limit, target vehicle speed, dimensions,etc.) associated with locations of the vehicle; and/or any othersuitable information.

The map is preferably a high definition map but can additionally oralternatively be or include any other suitable maps. The map is furtherpreferably hand labeled with any or all of the information describedabove, which can be enabled, for instance, by the small ODD and fixedroute approach to preferred variations of the system and method. Thisfunctions to produce a highly accurate, safe, and dependable map withwhich to transition between vehicle contexts. In variations ofconventional systems and methods implementing geofenced ODDs (e.g., asshown in FIG. 6), for instance, the number of potential routes availableto the vehicle is significant, thereby causing: more required switchesbetween contexts (e.g., increasing the chance of incorrectly selecting acontext); increased requirements for storing and switching between alarge number of fully learned models (e.g., resulting in acomputationally crippling problem); and/or any other effects.

The contexts are preferably assigned to one or more particular regionsin the map (e.g., hard-coded into the map, soft-coded into the map,etc.), such that a particular context relevant to the agent (e.g.,context in which agent is located, context in which the agent is aboutto be located, context that agent has departed, etc.) can be determined(e.g., with one or more inputs received in S210 such as pose informationof the autonomous agent) in S220.

The contexts are preferably assigned to locations and/or regions withinthe map. Each location and/or region in the map can be assigned any orall of: a single context; multiple contexts (e.g., indicating anintersection of multiple routes, wherein a single context is selectedbased on additional information such as any or all of the inputsreceived in S210, etc.); no context (e.g., indicating a location and/orregion not on a fixed route option for the autonomous agent); and/or anycombination of contexts. The particular context(s) assigned to thelocation and/or region are preferably determined based on the staticenvironment at that location and/or within that region, such as any orall of: features of the roadway within that region (e.g., number oflanes, highway vs. residential road, one-way vs. two-way, dirt and/orgravel vs. asphalt, curvature, shoulder vs. no shoulder, etc.);landmarks and/or features within that region (e.g., parking lot,roundabout, etc.); a type of zone associated with that location and/orregion (e.g., school zone, construction zone, hospital zone, residentialzone, etc.); a type of dynamic objects encountered at the locationand/or region (e.g., pedestrians, bicycles, vehicles, animals, etc.);traffic parameters associated with that location and/or region (e.g.,speed limit, traffic sign types, height limits for semi trucks, etc.);and/or any other environmental information.

In a first set of variations, the map is a high-definition map withhardcoded contexts. In specific example, the map is a geo-location filewith semantic annotations of context for variations points and/or areas.

Additionally or alternatively, the time of day at which a route is beingtaken and/or one or more dynamic (e.g., temporal) features can be takeninto account, such as any or all of: traffic patterns (e.g., at the timethat a fixed route is scheduled to and/or most likely to take place);weather conditions; lighting conditions; time-specific zone information(e.g., times at which school zone restrictions are enforced); and/or anyother suitable information.

In some variations, dynamic objects (e.g., surrounding vehicles,pedestrians, animals, moving objects, etc.) and/or non-permanent objectsor environments (e.g., construction sites) are preferably accounted forwith a sensor system (rather than the map), wherein in an event that thecontext is not recognizable and/or otherwise affected based on thedynamic object (as calculated in an uncertainty estimate), a fallbackmotion planner can be triggered. Additionally or alternatively, maps canbe dynamically and/or iteratively produced to account for any or all ofthe features and/or objects.

Additionally or alternatively, the context assignments can be determinedbased on route information (e.g., fixed route information), such as aset of fixed routes prescribed for the autonomous agent to take. Theroute information can include any or all of: a starting location of theroute, a destination of the route, a directionality of the autonomousagent along the route, and/or any other information. In variationsinvolving fixed routes, for instance, the contexts assigned to the mapare preferably selected based on the vehicle's progression along theroute and the contexts that the vehicle would sequentially encounter indoing so. In specific examples, for instance, an intersection at whichthe agent is planned to pass straight through in a first fixed route maybe assigned a different context than the same intersection at which theagent is planned to turn right at in a second fixed route.

Additionally or alternatively, any or all of the contexts can bedetermined independently of a route and/or a fixed route.

Additionally or alternatively, one or more contexts identified in S220can be determined based on a map without assigned and/or prescribedcontexts, such as in variations in which the context is identified withone or more programmed processes and/or machine learning processes(e.g., as described in S220) based on a set of inputs (e.g., thosedescribed in S210). In some variations, for instance, a context isidentified with a context identifier module (e.g., as part of theagent's planning module, as part of the computing system, etc.), such asthat shown in FIG. 13, wherein the context identifier module can receiveinputs from the map such as road infrastructure information (e.g.,location of, size of, parameters associated with, etc.) and/or staticenvironment features, which can individually and/or collectivelyinclude, but is not limited to, any or all of: road signs, lane lines,buildings, railroad tracks, bus routes, and/or other infrastructureinformation.

The map can optionally include (e.g., assign, prescribe, etc.) one ormore transition zones which are arranged between different contexts, andcan indicate, for instance, a change in context (e.g., along a fixedroute, along a dynamically determined route, etc.), thereby enabling aswitching of contexts to occur smoothly (e.g., by defining an actionspace). Assigning transition zones can function, for instance, to definean action space subsequently in the method which smoothly transitionsthe vehicle from one context to the next (e.g., preventing theavailability of certain actions, prescribing that the agent maintain hisor her lane, preventing a turn, etc.). The transition zones can be anyor all of: overlapping with (e.g., partially overlapping with, fullyoverlapping with, etc.) one or more contexts; non-overlapping with oneor more contexts; and/or any combination of overlapping andnon-overlapping. Additionally or alternatively, the transition zones canbe contexts themselves; the method can be performed in absence oflabeled transition zones (e.g., by anticipating the subsequent context);and/or be otherwise performed.

In a first variation, S205 includes hand labeling a high definition mapto prescribe a set of contexts, further preferably a series of contexts,for at least a set of fixed routes available to an autonomous agent.S205 further preferably includes hand labeling a set of static objectsand/or road features associated with the routes. In specific examples,each region in the map is assigned a single context (e.g., for allroutes, for a specific route, etc.). Additionally or alternatively, eachregion in the map can be assigned multiple contexts (e.g., foroverlapping routes, to be selected from, etc.).

In a second variation, S205 includes labeling a map without prescribedcontexts, wherein the context is determined subsequently in the method200 (e.g., with one or more machine learning models). In specificexamples, the map is labeled with road features and/or landmarks, whichare subsequently used (e.g., with sensor information) to determine acontext.

Additionally or alternatively, S205 can be performed in another suitableway or the method 200 can be performed in absence of S205.

4.2 Method—Receiving a Set of Inputs S210

The method 200 includes receiving a set of inputs S210, which functionsto receive information with which to determine a context for theautonomous agent. Additionally or alternatively, S210 can function toreceive information with which to perform any suitable processes of themethod (e.g., determine an action and/or behavior, determine atrajectory of the agent, etc.). Additionally or alternatively, S210 caninclude determining (e.g., calculating) any or all of the set of inputs,combining inputs (e.g., in one or more sensor fusion processes),processing and/or preprocessing the set of inputs, and/or any othersuitable processes.

S210 is preferably performed throughout the method 200, such as any orall of: continuously, at a predetermined frequency, at random intervals,prior to each of a set of processes of the method 200, and/or at anyother suitable times. S210 can additionally or alternatively beperformed in response to a trigger (e.g., based on the map, based onsensor information, etc.), at random intervals, and/or at any othersuitable time(s) during the method 200.

The set of inputs received in S210 preferably includes sensorinformation collected at a sensor subsystem of the autonomous agent,such as any or all of: a sensor system onboard the autonomous agent, asensor system remote from the autonomous agent, and/or a sensor systemin communication with the autonomous agent and/or a computing system(e.g., onboard computing system, remote computing system, etc.) of theautonomous agent. Additionally or alternatively, the sensor informationcan be collected from any other suitable sensor(s) and/or combination ofsensors, S210 can be performed in absence of collecting sensor inputs,and/or S210 can be performed in any other suitable way(s).

The sensor information preferably includes location informationassociated with the autonomous agent, such as any or all of: position,orientation (e.g., heading angle), pose, geographical location (e.g.,using global positioning system [GPS] coordinates, using othercoordinates, etc.), location within a map, and/or any other suitablelocation information. In preferred variations, for instance, S210includes receiving pose information from a localization module of thesensor subsystem, wherein the localization module includes any or allof: GPS sensors, IMUs, LIDAR sensors, cameras, and/or any other sensors(e.g., as described above). Additionally or alternatively, any othersensor information can be received from any suitable sensors.

The sensor information can additionally or alternatively include motioninformation and/or other dynamic information associated with theautonomous agent, such as, but not limited to, any or all of:velocity/speed, acceleration, and/or any other suitable information.

The sensor information further preferably includes location informationand/or motion information associated with one or more dynamic objects inan environment of the autonomous agent, such as any or all of thelocation information described above, location information relative tothe autonomous agent, motion information of the dynamic objects,predicted information (e.g., predicted trajectory), historicalinformation (e.g., historical trajectory), and/or any other suitableinformation. The dynamic objects can include, but are not limited to,any or all of: other vehicles (e.g., autonomous vehicles, non-autonomousvehicles, 4-wheeled vehicles, 2-wheeled vehicles such as bicycles,etc.), pedestrians (e.g., walking, running, rollerblading,skateboarding, etc.), animals, and/or any other moving objects (e.g.,ball rolling across street, rolling shopping cart, etc.). Additionallyor alternatively, the sensor information can include any otherinformation associated with one or more dynamic objects, such as thesize of the dynamic objects, an identification of the type of object,other suitable information, and/or the information collected in S210 canbe collected in absence of dynamic object information.

The sensor information can optionally include location informationand/or other information associated with one or more static objects(e.g., stationary pedestrians, road infrastructure, construction siteand/or construction equipment, barricade(s), traffic cone(s), parkedvehicles, etc.) in an environment of the autonomous agent, such as anyor all of the information described above (e.g., identification ofobject type, etc.). Additionally or alternatively, the sensorinformation can include any other information associated with one ormore static objects and/or the information collected in S210 can becollected in absence of static object information.

The set of inputs received in S210 further preferably includes the mapand/or any information determined from (e.g., determined based on,derived from, included in, etc.) the map, such as any or all of theinformation described above in S205. In some variations, this includesone or more contexts (and/or transition zones) selected based on (e.g.,predetermined/assigned to) a region/location of the autonomous agent(e.g., as determined based on sensor information as described above). Inadditional or alternative variations, the map information includes anyor all of: road infrastructure information and/or other staticenvironment information, route information, and/or any other suitableinformation.

Information associated with the map can optionally be determined basedon other information received in S210, such as any or all of the sensorinformation received at one or more sensor systems. For instance,location information (e.g., current pose, current position, currentgeographical location, etc.) associated with the autonomous agent can beused to locate the agent within the map (e.g., determine its positionwithin a hand labeled map), which is used to determine the mapinformation relevant to the autonomous agent. In a first set ofvariations, this information is a particular context selected (e.g.,assigned to) based on the location of (e.g., a region including thelocation, a stretch of road on which the agent is located, a particularintersection in which the agent is located, etc.) the autonomous agent.In a second set of variations, the location of the autonomous agent incomparison with the map includes information associated with the roadinfrastructure (e.g., road signs, lane lines, buildings, etc.), which isused, preferably along with other inputs (e.g., static environmentfeatures, static object information, autonomous agent vehicle state,dynamic environment features, etc.) to determine a context (e.g., usingone or more learning-based models, using a pattern recognition and/orclassification model, etc.) for the autonomous agent.

Additionally or alternatively, the map information can include any otherinformation (e.g., a set of possible contexts, a set of parametersand/or weights for an algorithm and/or model, etc.) and/or be receivedin any other way, such as, but not limited to, any or all of:independently of other information received in S210, concurrently withother information received in S210, prior to other information receivedin S210, subsequent to other information received in S210, multipletimes in S210, and/or at any other suitable time(s).

The set of inputs can optionally include a route and/or associated routeinformation (e.g., route identifier of a fixed route, agent'sprogression through route, etc.) assigned to and/or being traversed bythe agent (e.g., fixed route selected for the agent, dynamic route beingtraveled by the agent, predicted route for the agent, etc.), which canfunction for instance, to select information from one or more mapsand/or to select a particular map (e.g., a route-specific map). The mapinformation selected based on route can include, but is not limited to,any or all of: a current context, a future context (e.g., next contextin a fixed route), a transition zone, and/or any other suitableinformation from a map. In variations in which a context is selectedbased on a map, route information can be used to select the appropriatecontext for the particular route in regions in which multiple routes areoverlapping (e.g., at an intersection which multiple routes passthrough). The route information (e.g., previous contexts of route,historical information, fixed route identifier, destination, startingpoint, directionality of route, etc.) can be used, for instance, toselect the proper context from multiple context options. Additionally oralternatively, the route information can be used in other processes ofthe method 200 (e.g., in defining an action space based on the context,in selecting a behavior from the action space, in determining theagent's trajectory, etc.), any other suitable information can bedetermined based on route information, the set of inputs can becollected independently of and/or in absence of route information,and/or S210 can be performed in any other suitable ways.

The set of inputs S210 can additionally or alternatively include anyother suitable information, such as, but not limited to, any or all of:a state (e.g., operational state, driving state, etc.) of the autonomousagent, a trajectory of the agent, a set of control commands for theagent, historical information associated with the agent and/or anenvironment of the agent, predicted information associated with theagent and/or the environment (e.g., predicted trajectories of dynamicobjects), and/or any other suitable information and/or inputs.

In a first set of variations (e.g., as shown in FIG. 12), S210 includesreceiving a map specifying a set of assigned contexts for an agent;optionally a route (e.g., fixed route) of the agent; and sensorinformation from a set of sensors onboard the autonomous agent, whereinthe sensor information includes at least a pose of the autonomous agent,wherein the pose and optionally the route are used to select a contextfor the agent based on the map. Additionally or alternatively, S210 caninclude receiving any other suitable inputs.

In a set of specific examples, S210 includes receiving a selected routefor the vehicle, a high definition hand-labeled map specifying a contextfor each portion of the route (and optionally other routes), and sensorinformation including at least location information associated with thevehicle and optionally any or all of: motion information, objectinformation (e.g., dynamic object information, static objectinformation, etc.), and/or any other suitable information.

In a second set of variations, S210 includes receiving a map includinginformation associated with road infrastructure, such as the roadinfrastructure along one or more routes of the agent (e.g., a fixedroute of the agent) and optionally any or all of: other map information(e.g., speed limit information, traffic laws, etc.); a state of theagent, equivalently referred to herein as an ego state (e.g., asdetermined by a sensor subsystem; pose, velocity, and acceleration;etc.); static environment features and/or information; dynamicenvironment features and/or information; sensor information; and/or anyother suitable information, wherein the context is determined with oneor more models and/or algorithms (e.g., convolutional neural networks[CNNs], recurrent neural networks [RNNs], support-vector machines[SVMs], etc.).

In a set of specific examples, a context identifier module, whichincludes one or more deep learning models, receives as input a map andoptionally route information for the agent, a state of the agent, staticenvironment features, and dynamic environment features, with which thecontext identifier module determines a context for the agent.

In a third set of variations, S210 includes receiving a set of multiplepossible contexts for the agent based on a map, wherein a context of theset of multiple contexts is determined based on other inputs received inS210.

Additionally or alternatively, S210 can include any other suitableprocesses.

4.3 Method—Determining a Context Associated with the Autonomous AgentBased on the Set of Inputs S220

The method 200 includes determining a context associated with theautonomous agent S220, which functions to specify the context in whichthe autonomous agent is operating, and can further function to: select alearning module based on the context (e.g., according to a 1:1 mapping),define and/or a limit a set of behaviors or actions available to theagent, specify particular parameters (e.g., creep distance) associatedwith the behaviors and/or actions, reduce and/or minimize the amount ofdata required to training the learning modules, and/or perform any othersuitable function(s).

S220 is preferably performed in response to (e.g., after, based on,etc.) S210, but can additionally or alternatively be performed as partof S210 and/or concurrently with S210, in place of S210, in absence ofS210, multiple times throughout the method, and/or at any other time(s)during the method 200. Further additionally or alternatively, the method200 can be performed in absence of S220.

A context refers to a high level driving environment of the agent, whichcan inform and restrict the vehicle's decision at any given time and/orrange of times. The context can include and/or define and/or bedetermined based on any or all of: a region type of the vehicle (e.g.,residential, non-residential, highway, school, commercial, parking lot,etc.); a lane feature and/or other infrastructure feature of the roadthe vehicle is traversing (e.g., number of lanes, one-way road, two-wayroad, intersection, two-way stop and/or intersection, three-way stopand/or intersection, four-way stop and/or intersection, lanes in aroundabout, etc.); a proximity to one or more static objects and/orenvironmental features (e.g., particular building, body of water,railroad track, parking lot, shoulder, region in which the agent canpull over/pull off to the side of a road, etc.); a proximity a parameterassociated with the location (e.g., speed limit, speed limit above apredetermined threshold, speed limit below a predetermined threshold,etc.); road markings (e.g., yellow lane, white lane, dotted lane line,solid lane line, etc.); and/or any other suitable information.

Examples of contexts can include, but are not limited to, any or all of:a two-way, two-lane residential road (e.g., in which the agent cannotchange contexts due to road geometry as shown in FIG. 4); a two-way,two-lane non-residential road; a multi-lane highway (e.g., in which theagent can learn it is less likely to see pedestrians); a one-way,single-lane road; a one-way, two-lane road; a one-way road with “n”number (e.g., 1, 2, 3, 4, 5, 6, greater than 6, etc.) of lanes; atwo-way road with “n” number (e.g., 1, 2, 3, 4, 5, 6, greater than 6,etc.) of lanes; a single lane road in a parking lot; a single lane roadwith a yellow boundary on the side; a multi-lane fast moving road (e.g.,having a speed above a predetermined threshold); an on ramp of ahighway; an off-ramp of a highway; regions connecting to roads (e.g.,parking lot, driveway, etc.); and/or any other suitable contexts.

S220 is preferably performed based on a set of inputs received in S210,but can additionally or alternatively be performed based on any othersuitable information.

S220 is preferably performed by reading a map (e.g., as described above)to determine the context assigned to a point and/or area correspondingto the location of the agent (e.g., pose) and/or a route of the agent.As described previously, each point and/or region can be any or all of:associated with at most 1 context (in a 1:1 mapping), associated with atmost 1 context per route (e.g., wherein a fixed route assignment of theagent is used to select the proper context), associated with multiplecontexts (e.g., which are ranked, prioritized, selected from based onother inputs received in S210), and/or otherwise associated. In specificexamples, the context assignments are hard-coded into the map.Alternatively, the context assignments can be soft-coded and/orotherwise assigned.

Additionally or alternatively, S220 can be performed with any number ofalgorithms, models (e.g., machine learning models, deep learning models,supervised learning models, unsupervised learning models,semi-supervised learning models, statistical models, pattern recognitionmodels, etc.), finite state machines (FSMs), processes (e.g.,traditionally programmed process), decision trees, and/or equations.

In some variations, for instance, one or more machine learning models,such as, but not limited to: one or more neural networks (e.g., CNNs,RNNs, etc.); SVMs; and/or any other suitable models, are implemented todetermine a context (e.g., as a context identifier module of FIG. 13) ofthe agent based on any or all of the inputs in S210.

Determining a context can optionally include switching between contexts,which preferably includes determining a transition between a currentcontext and a future context. The transition can be in the form of anyor all of: a transition zone (e.g., as described above) prescribed inthe map; a change in contexts; a transition action/behavior (e.g., lanechanging, merging, exiting a highway, etc.); a transition trajectory(e.g., trajectory taken by the vehicle to change from a 1^(st) lane intoa 2^(nd) lane, etc.); and/or any can prescribe any other motion for thevehicle. The transition is preferably determined (e.g., prescribed)based on the map (e.g., as indicated as a transition zone, as indicatedthrough distinct adjacent contexts in the map, as indicated throughdistinct sequential contexts in a fixed route, etc.), but canadditionally or alternatively be dynamically determined, determined witha trajectory planner, determined based on sensor information, and/orotherwise determined.

S220 can optionally include selecting a scenario based on the context,which functions to further specify the context, such as based on any orall of the information described above (e.g., speed limit, sensorinformation of objects surrounding vehicle, etc.). Examples of scenariosfor a first context (e.g., a two-way residential road) include, but arenot limited to, any or all of: a right turn opportunity; an addition ofa right turn lane; a stop sign; a traffic light; a yield sign; acrosswalk; a speed bump; and/or any other scenarios. Examples ofscenarios for a second context (e.g., a multi-lane highway) include, butare not limited to, any or all of: lane changing; merging; overtaking aslow moving vehicle; and/or any other scenarios. In some variations, forinstance, the context triggers the selection of a model and/or algorithm(e.g., a highly-tuned, context-aware custom inverse reinforcementlearning (IRL) algorithm), which makes high-level scenario selection andcalls a scenario-specific learning module (e.g., as described below) toselect an action of the vehicle. Additionally or alternatively, anyother suitable algorithms or processes for selecting a scenario can beimplemented, an action can be selected in absence of a scenario, acontext can be used to select another parameter, and/or S220 can beotherwise performed.

In a first set of variations, S220 includes selecting a context based ona location and/or orientation of the vehicle (e.g., pose), a labeledmap, and optionally any or all of the other information received inS210, wherein the context informs how the remaining processes of themethod are performed. Optionally, the context then triggers theselection of a particular scenario (e.g., based on a context-specificIRL algorithm).

In a set of specific examples (e.g., as shown in FIG. 14), a labeled mapindicates a sequential series of contexts for each of set of routes(e.g., C1 to C2 to C3 for Route 1; C1′ to C2′ to C3′ to C4′ to C5′ forRoute 2; etc.), wherein in an event that the routes overlap in aparticular section and/or point (e.g., intersection), a route assignment(e.g., Route 1 vs. Route 2) and/or directionality of the agent (e.g.,West vs. East) can be used to select the proper context. The map canoptionally further include transition zones (e.g., having a non-zerosize, having a size of zero and indicating an immediate transition,etc.) between adjacent regions of different context(s) and/or any otherinformation.

In an additional or alternative set of specific examples (e.g., as shownin FIG. 15), a labeled map assigns a context to each of a set of regionsof a labeled map, wherein a location of the agent within the region (andoptionally a route assignment and/or directionality of the agent)determines the context for the agent. The map can optionally furtherinclude transition zones (e.g., having a non-zero size, having a size ofzero and indicating an immediate transition, etc.) between adjacentregions of different context(s) and/or any other information.

In a second set of variations, S220 includes identifying a contextassociated with the agent with a context identifier module, the contextidentifier module including one or more trained models (e.g., machinelearning model(s), deep learning model(s), etc.), which receives asinput any or all of the information received in S210.

Additionally or alternatively, S220 can include any other suitableprocesses and/or be performed in any other suitable ways.

4.4 Method—Selecting a Learning Module Based on the Context and/orScenario S230

The method 200 can include selecting a learning module based on thecontext and/or scenario S230, which functions to enable an action(equivalently referred to as a behavior) to be determined for the agentwhich takes into account the particular context (and optionallyscenario) of the vehicle. S230 can additionally or alternativelyfunction to define an action space available to the agent, inform atrajectory of the agent as determined by a trajectory planner, eliminateone or more actions from consideration by the agent (e.g., minimize anumber of available actions to an agent), and/or can perform any othersuitable functions.

S230 is preferably performed in response to (e.g., after, based on,etc.) S220, but can additionally or alternatively be performed as partof S220 and/or concurrently with S220, in place of S220, in absence ofS220, in response to S210, multiple times throughout the method, and/orat any other time(s) during the method 200. Further additionally oralternatively, the method 200 can be performed in absence of S230.

S230 preferably includes selecting a learning module (equivalentlyreferred to herein as a context-aware learning agent or a deep decisionnetwork) which includes a set of machine learning (e.g., deep learning)models and/or algorithms, wherein the learning module is trained basedon data associated with that particular context. This functions todivide a large amount of data from all possible contexts into a set ofmanageable amounts, which cover all or nearly all of the situations theagent would encounter in that context.

Each context is preferably mapped in a 1:1 fashion to a learning module.Additionally or alternatively, a context can be associated with multiplelearning modules (e.g., where results from multiple modules areaggregated, where a single learning module is then selected, etc.); alearning module can be associated with multiple contexts; and/or thecontexts and learning modules can be otherwise mapped.

S230 can optionally include receiving a set of inputs, which can includeany or all of the set of inputs described above, a different and/oradditional set of inputs, and/or any other suitable inputs. In a set ofvariations, for instance, S230 includes receiving any or all of: theinputs described above; the context and/or scenario of the agent; theset of vehicles and/or other dynamic objects surrounding the vehicle,the predicted paths (e.g., where will it be in lane and in which lane,etc.) of the dynamic objects, static objects surrounding the agent;uncertainty values (e.g., of the predicted paths); routing informationassociated with the agent; and/or any other suitable inputs.

In some variations (e.g., as shown in FIG. 16), an environmentalrepresentation of the agent is received with the context at a deepdecision network selected based on the context. In an example of thisshown in FIG. 16, the environmental representation (referred to as afull environmental representation) includes a latent spacerepresentation of a set of inputs (e.g., as described in S210,additional or alternative to those described in S210, etc.), the set ofinputs including any or all of: a state of the agent (equivalentlyreferred to herein as an ego vehicle state), one or more maps, routinginformation (e.g., a selected fixed route, parameters associated with aselected fixed route, etc.), dynamic object information/features, staticobject information/features, and/or any other suitable information. Oneor more models (e.g., machine learning models, deep learning models,RNNs, etc.) and/or processes and/or algorithms can optionally be used toprocess any or all of these inputs (e.g., to determine a latent spacerepresentation, to determine another output, to simplify the input(s) tothe deep decision network, etc.). In specific examples (e.g., as shownin FIG. 16), for instance, a first neural network (e.g., one or moreRNNs, one or more CNNs, a combination of RNNs and CNNs, OrderIndependent Representation in FIG. 16, etc.) is used process objectfeatures (e.g., dynamic, static, both, etc.) in the environment of theagent, which can function, for instance to produce an intermediaterepresentation (e.g., abstraction) of object features. In specificexamples, the first neural network is a CNN including a set of one ormore convolution and/or pooling layers. Additionally or alternatively,any other suitable models can be used. This representation canoptionally be combined with (e.g., appended with in a vector and/ormatrix) other environmental information/inputs (e.g., route information,map, agent/ego pose, etc.) and/or outputs from other models, such as asecond neural network (e.g., an RNN, a CNN, Ego RNN in FIG. 16, etc.)which processes the ego vehicle state. Any or all of this informationcan then be fed into a latent space network (e.g., RNN, CNN, etc.),which is used to determine the environmental representation based on theoutputs of these other networks and optionally map information, routinginformation, and/or any other suitable inputs. The latent space networkis preferably in the form of and/or includes one or more autoencoders(with one or more encoders, code, and decoders), but can additionally oralternatively include any or all of: other unsupervised learning models,supervised learning models, semi-supervised learning models, and/or anyother suitable neural networks and/or models. In specific examples, thelatent space network is a fully-connected feedforward neural network.The output of the latent space representation, which is preferably alow-dimensional features vector (but can additionally or alternativelybe any other output) is preferably then used as an input to the learningmodule.

Additionally or alternatively, any other suitable models and/oralgorithms can be implemented, the input and/or outputs can be differentfor these models and/or algorithms, and/or any suitable modelarchitecture can be implemented.

The learning module is preferably in the form of a machine learningmodel, further preferably in the form of one or more neural networksand/or models (e.g., convolutional neural network [CNN], inversereinforcement learning [IRL] model, reinforcement learning [RL] model,imitation learning [IL] model, etc.) trained for a particular contextand/or contexts, but can additionally or alternatively include any othersuitable models, algorithms, decision trees, lookup tables, and/or othertools.

In preferred variations, each of the learning modules is a neuralnetwork, further preferably a deep Q-learning network (e.g., IRLalgorithm/network), wherein the number of layers (e.g., hidden layers)of the neural network can vary for different contexts and/or actions(e.g., between 3-8 layers, 3 or less layers, 8 or more layers, between 2and 10 layers, between 1 and 15 layers, etc.). Additionally oralternatively, any other suitable networks, algorithms, and/or modelscan be used in the learning module(s), such as, but not limited to, anyor all of: policy gradient methods, finite state machines [FSMs],probabilistic methods (e.g., Partially Observable Markov DecisionProcess [POMDP]), imitation learning [IL], RL or variations of IRL,and/or any other suitable models and/or networks and/or algorithms. Eachof the learning modules is preferably the same type of neural network(e.g., with different numbers of layers) and/or algorithm and/or model,but can alternatively be different.

Each of the learning modules is preferably trained based on dataoccurring within the particular context type or context types associatedwith the learning module and optionally further trained based on dataoccurring within one or more fixed routes which pass through thecontext. In some variations, for instance, a single learning moduleapplies to a particular context type, wherein the single learning moduleis trained based on different versions of that context. In othervariations, a single learning module applies to a particular contextwithin a particular route, wherein the single learning module is trainedbased on data associated with that particular context in the particularfixed route. Additionally or alternatively, the learning module(s) canbe trained on any suitable data.

Each of the learning modules is further preferably trained with inversereinforcement learning, which functions to determine a reward functionand/or an optimal driving policy for each of the context-aware learningagents. The output of this training is further preferably a compactfully-connected network model that represents the reward function and anoptimal policy for each learning module. Additionally or alternatively,the learning modules can be otherwise suitably trained and/orimplemented.

In a first variation, S230 includes selecting a context-aware learningagent (equivalently referred to herein as a context-aware learningmodule) based on a determined context of the agent, wherein a singlecontext-aware learning agent is assigned to each context. Thecontext-aware learning agent is preferably trained with an inversereinforcement learning technique, but can additionally or alternativelybe otherwise trained.

In a second variation, S230 includes selecting from multiplecontext-aware learning agents assigned to and/or available to aparticular context, wherein the particular context-aware learning agentis selected based on any or all of: machine learning, a decision tree,statistical methods, an algorithm, and/or with any other suitabletool(s).

Additionally or alternatively, any other suitable learning modules canbe selected, used, and/or trained.

4.5 Method—Defining an Action Space Based on the Learning Module S240and Selecting an Action from the Action Space S250

The method 200 can include defining an action space based on thelearning module S240, which functions to define a set of actions(equivalently referred to herein as behaviors) available to the agent inlight of the vehicle's context and/or environment. Additionally oralternatively, S240 can function to minimize a number of availableactions to the agent as informed by the context, which functions tosimplify the process (e.g., reduce the time, prevent selection of anincompatible action, etc.) required to select an action for the vehicle.The method 200 can optionally additionally or alternatively includeselecting an action from the action space S250, which functions todetermine a next behavior (e.g., switching and/or transitioning to adifferent behavior than current behavior, maintaining a currentbehavior, etc.) of the vehicle.

S240 and/or S250 are preferably performed in response to (e.g., after,based on, etc.) S230, but can additionally or alternatively be performedas part of S230 and/or concurrently with S230, in place of S230, inabsence of S230, in response to S220 and/or S210, multiple timesthroughout the method, and/or at any other time(s) during the method200. Further additionally or alternatively, the method 200 can beperformed in absence of S240 and/or S250.

The action space and/or action is preferably produced as an output(e.g., intermediate output, final output, etc.) of the learning module;additionally or alternatively, the learning module can produce any othersuitable outputs. In preferred variations, a determination of thecontext and processing based on this context (e.g., the specificlearning module) allows the action space to be relatively small (e.g.,relative to all available actions).

The actions can include, but are not limited to, any or all of:maintaining a lane, changing lanes, turning (e.g., turning right,turning left, performing a U-turn, etc.), merging, creeping, following avehicle in front of the agent, parking in a lot, pulling over, nudging,passing a vehicle, and/or any other suitable actions such as usualdriving actions for human-operated and/or autonomous vehicles.

Each action is preferably associated with a set of parameters, which aredetermined based on the particular context of the agent and optionallyany other suitable inputs (e.g., sensor information). This highlights abenefit of this architecture, which enables various parameter values tobe associated with an action, wherein the context specifies theparticular value, thereby enabling the action learned for differentcontexts to be different yet predictable. In contrast, in conventionalmethods where the method is entirely programmed, for instance, one wouldneed to either generalize the parameter (e.g., creep distance) to havean overly conservative value or program multiple values for differentcases; and in methods including only learning based approaches, thiswould lead to an oversimplification of the action across cases, whichcould result in unpredictable agent behavior at times (e.g., roboticbehavior, the production of an infeasible trajectory, etc.).

For preferred variations of this method, the extra information andrestriction from the context type can reduce the amount of data that isneeded to train the different learning approaches and better tune theagent to a specific context to increase accuracy and confidence.

In preferred variations, an output layer of each deep decision networkis a softmax layer where the number of output nodes is the number ofavailable actions. Additionally or alternatively, an action space and/oravailable actions can be determined in any other suitable way(s).

In a specific example, a multi-lane highway context produces acorresponding action space including: maintaining speed, lane changeleft, and lane change right. In contrast, a different context such as aresidential road produces actions such as those in the highway contextand additional actions such as stop, yield, creep, left turn, and rightturn.

In additional or alternative variations, an output layer (e.g., linearoutput layer) can be used to generate an embedding (e.g., a vector, avector of real numbers, etc.) for the action, wherein the embeddingcould be matched to stored embeddings associated with particular actions(e.g., at a lookup table). In specific examples, for instance, a lengthand/or angle of an embedding vector produced by an output layer can beused to match it to a vector associated with a particular action.

Selecting an action in S250 can be performed by the context-awarelearning agent, performed with another model and/or algorithm and/orprocess, determined based on other information (e.g., any or all of theset of inputs from S210, based on the particular route, based on a nextcontext in the map, etc.), and/or otherwise determined.

In preferred variations, the action is produced as an output (e.g.,single output, multiple outputs, etc.) of the context-aware learningagent.

In additional or alternative variations, the action can be determinedbased on a state machine or other rule-based method for choosing anaction based on context.

In a first variation, the context of the agent is determined from a mapto be a one-lane residential road (e.g., in which the agent cannotchange contexts due to road geometry as shown in FIG. 4). A set ofactions determined for this context can include, for instance:maintaining speed, creeping, left turning, right turning, and yielding.For creeping (e.g., as shown in FIG. 5), a major parameter is creepdistance, which refers to the distance the agent should creep forwardwith extra caution (e.g., before deciding to merge). For instance,humans tend to creep at a stop sign or before merging on a highway tocautiously gauge any oncoming traffic and pace the speed of the vehicleto merge without collisions or annoyance to road users. Depending on theparticular context and optionally action, the value of this parameter isdifferent. In specific examples (e.g., as shown in FIG. 5), for thecontext of a parking lot and the action of turning right and/or stoppingat a stop sign, the creep distance is 2 meters, whereas for the contextof a multi lane highway and the action of merging, the creep distance is17 meters.

In a second variation, the context of the agent is determined to be amulti-lane highway in which the agent can learn (e.g., in the learningmodule) it is less likely to see pedestrians. The actions of the actionspace can include, for instance: lane swap left, lane swap right,maintain speed, and stop.

4.6 Method—Planning a Trajectory Based on the Action S260

The method 200 can include planning a trajectory based on the actionS260, which functions to enable the agent to perform the selected action(e.g., as described above).

The trajectory preferably specifies the set of locations and associatedspeeds for the agent to be at in order to perform the selected action.The trajectory is preferably generated based on one of a set oftrajectory learning modules (e.g., different than the learning moduledescribed above, separate from but including the same and/or a similararchitecture as the learning modules described above, etc.), but canadditionally or alternatively be generated with any other suitabletools, programmed or learned.

S260 can optionally additionally or alternatively include any or all of:validating the trajectory, implementing a fallback mechanism, operatingthe vehicle according to a trajectory, determining control commands withwhich to operate the vehicle based on a trajectory, and/or any othersuitable output.

5. Variations

In a first variation of the method 200, the method includes: receiving aset of inputs S210, the set of inputs including at least map (e.g., highdefinition hand-labeled map, map labeled in an automated fashion, maplabeled both manually and in an automated fashion, etc.), a vehiclelocation (e.g., pose), and optionally a route planned for the vehicle;determining a context prescribed by the map based on the location of theagent; selecting a learning module including a neural network based onthe context; defining an action space including a set of actionsavailable to the agent with the learning module; and selecting an actionfrom the action space (e.g., with the learning module). Additionally oralternatively, the method 200 can include any other suitable processes(e.g., determining a trajectory based on the action).

In a specific example, the method 200 includes: receiving a set ofinputs, wherein the set of inputs includes a hand labeled, highdefinition map prescribing a set of contexts, further preferably aseries of contexts, for at least a fixed route of the autonomous agent,wherein the set of inputs further includes sensor information from a setof sensors onboard the autonomous agent, wherein the sensor informationincludes at least a pose of the autonomous agent, wherein the pose andoptionally the route are used to select a context for the agent based onthe map, and optionally any other suitable inputs; selecting a contextbased on a location and/or orientation of the vehicle (e.g., pose), thelabeled map, and optionally any or all of the other informationreceived, wherein the context informs how the remaining processes of themethod are performed; selecting a context-aware learning agent based onthe context, wherein a single context-aware learning agent is assignedto each context and trained (e.g., with an inverse reinforcementlearning model); defining an action space and selecting an action basedon the learning module; and determining a trajectory for the vehiclebased on the action space and/or action.

In a specific implementation as shown in FIGS. 7A-7D, the set of inputsincludes receiving a route as shown in FIG. 7A; determining a firstcontext shown in FIG. 7B based on the route and the map, wherein fromthe start of the trip, the vehicle is located in a parking lot typecontext with a single lane of traffic and expected heavy foot traffic.This foot traffic is usually localized to sidewalk however in thiscontext pedestrian cutoff events are highly probable. To handle thiscontext, the learning agent is optimized to be significantly more awareof pedestrians and their actions and as such the reward function istuned to achieve this. The available actions to this agent are:maintaining a predetermined speed (e.g., speed bounded by any leadingdynamic objects in agent's path and the speed limit of the currentroad); yielding behavior relative to a set of dynamic objects (e.g.,which may have a precedence over the agent); staying stopped in thecurrent location; and providing a stopping location which the egovehicle must stop by. The second context that the vehicle encountersalong this route, shown in FIG. 7C, is the context of a single laneresidential road. This context is quite large including handling singlelane traffic lights and stop sign intersection, pedestrian's crossingand right-hand turns. The decisions required to handle this context arevery similar to that of the first context, the main difference beingwhat the algorithm is trained on and that the reward function is tunedfor two different sets of behaviors. The final context encountered bythe agent, shown in FIG. 7D, is a multi-lane residential context, whichincludes everything that the single lane context needs to handle but inmultiple lanes. Thus it needs to understand how to switch lanes, and howto handle turning on multi lane intersection. To handle this, inaddition to the actions from the second context, it also contains thefollowing two actions in its action space: changing to the left lane(when safe to do so) and changing to the right lane (when safe to doso).

In a second variation of the method 200, the method includes: receivinga set of inputs S210, the set of inputs including at least map (e.g.,high definition hand-labeled map, map labeled in an automated fashion,map labeled both manually and in an automated fashion, etc.), a vehiclelocation (e.g., pose), and optionally a route planned for the vehicle;determining a context for the agent with a context identifier modulebased on the inputs; selecting a learning module including a neuralnetwork based on the context; defining an action space including a setof actions available to the agent with the learning module; andselecting an action from the action space (e.g., with the learningmodule). Additionally or alternatively, the method 200 can include anyother suitable processes (e.g., determining a trajectory based on theaction).

Additionally or alternatively, the method 200 can include any othersuitable processes and/or be performed in any suitable way(s).

Although omitted for conciseness, the preferred embodiments includeevery combination and permutation of the various system components andthe various method processes, wherein the method processes can beperformed in any suitable order, sequentially or concurrently.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

We claim:
 1. A method for operation of an autonomous vehicle along aroute, the method comprising: collecting a set of sensor inputs from aset of sensors onboard the autonomous vehicle; processing the set ofsensor inputs to determine a location associated with the autonomousvehicle; with a deep neural network, limiting a set of available actionsfor the vehicle based on the location to produce a limited set ofavailable actions; selecting an action from the limited set of availableactions; and operating the autonomous vehicle along the route based on atrajectory which executes the selected action.
 2. The method of claim 1,wherein the limited set of available actions is further determined basedon a predetermined map, wherein the predetermined map includes a set oflabels corresponding to each of a set of locations, the set of locationscomprising the location.
 3. The method of claim 1, wherein each of theset of routes is fixed.
 4. The method of claim 3, wherein the limitedset of available actions is further determined based on a map, whereinthe map includes the set of routes, wherein at least a portion of routesin the set of routes are overlapping relative to each other.
 5. Themethod of claim 4, wherein the limited set of available actions isfurther determined based on an orientation of the autonomous vehicle,the orientation of the vehicle determined based on processing the set ofsensor inputs.
 6. The method of claim 3, wherein each of the set ofroutes is limited to a distance of less than 100 miles.
 7. The method ofclaim 6, wherein each of the set of routes is limited to a distance ofless than 20 miles.
 8. The method of claim 6, wherein each of the set ofroutes is arranged between a distribution center and a retail store. 9.The method of claim 1, wherein the deep neural network comprises arecurrent neural network.
 10. The method of claim 1, wherein the limitedset of available actions is further determined based on a set of staticenvironmental features.
 11. The method of claim 10, wherein the limitedset of available actions is further determined based on a set of dynamicenvironmental features.
 12. A method for operation of an autonomousvehicle along a fixed route, the method comprising: receiving a set ofsensor inputs from a set of sensors onboard the autonomous vehicle;processing the set of sensor inputs to determine a location of theautonomous vehicle, wherein the location of the autonomous vehicle isalong the fixed route, the fixed route arranged between a distributioncenter and a retail store; with a neural network, determining a set ofavailable actions for the vehicle based on the location; selecting anaction from the set of available actions; and operating the autonomousvehicle along the fixed route based on a trajectory which executes theselected action.
 13. The method of claim 12, wherein the set ofavailable actions for the autonomous vehicle is further determined basedon the fixed route.
 14. The method of claim 13, wherein the set ofavailable actions for the autonomous vehicle is further determined basedon a map.
 15. The method of claim 14, wherein the map comprises thefixed route.
 16. The method of claim 12, wherein the neural networkcomprises a deep neural network.
 17. The method of claim 16, wherein thedeep neural network comprises a recurrent neural network.
 18. The methodof claim 12, further comprising determining the trajectory based on asecond neural network, the second neural network separate and distinctfrom the first neural network.
 19. The method of claim 12, wherein eachof the set of routes is limited to a distance of less than 100 miles.20. The method of claim 19, wherein each of the set of routes is limitedto a distance of less than 20 miles.